Constrained Formulations for Neural Network

In this paper, we formulate neural-network training as a constrained optimization problem instead of the traditional formulation based on unconstrained optimization. We show that constraints violated during a search provide additional force to help escape from local minima using our newly developed constrained simulated annealing (CSA) algorithm. We demonstrate the merits of our approach by training neural networks to solve the two-spiral problem. To enhance the search, we have developed a strategy to adjust the gain factor of the activation function. We show converged training results for networks with 4, 5, and 6 hidden units, respectively. Our work is the rst successful attempt to solve the two-spiral problem with 19 weights. Traditional supervised neural-network training is formulated as an unconstrained optimization problem of minimizing the sum of squared errors of the output over all training patterns: the number of training patterns, and w is a vector of weights of the neural network trained. In order for a neural network to generalize well to unseen patterns, we like it to have a small number of weights. Training is diicult in this case because the terrain modeled by (1) is often very rugged, and existing local-search algorithms may get stuck easily in deep local minima. Although global search can help escape from local minima, it has similar diiculties when the terrain is rugged. Instead of using an unconstrained formulation (1), we propose to formulate neural-network training as a constrained optimization problem that includes constraints on each training pattern. An unsatissed training pattern in a local minimum of the weight space may provide an additional force to guide a search out of the local minimum. The constrained formulation considered in this paper is: min w